Skip to content

Add support for significant figures (sig_figs) parameter - #273

Open
m-messer wants to merge 2 commits into
mainfrom
feature/sig_fig
Open

Add support for significant figures (sig_figs) parameter#273
m-messer wants to merge 2 commits into
mainfrom
feature/sig_fig

Conversation

@m-messer

@m-messer m-messer commented Aug 7, 2026

Copy link
Copy Markdown
Member

Problem

This was a requested feature (#270). At the moment, the evaluation function can only check a numeric answer using absolute or relative tolerance. Some questions need something more specific: they need to check that a student's answer is written to a required number of significant figures, not just that the value is close enough.

For example, if the correct answer is 92 and the question asks for 4 significant figures, 92.00 should be accepted but 92 should not, even though they represent the same number. Tolerance alone cannot tell these apart, since it only checks how close the value is, not how precisely it was written.

Changes

  • app/utility/expression_utilities.py: added four new functions.
    • round_to_sig_figs: rounds a number to a given number of significant figures.
    • split_numeric_string: checks that a response is written as a plain number (e.g. "92.00"), and splits it into its whole-number part, decimal part, and whether a decimal point was written.
    • count_sig_figs: counts the significant figures in those parts, following the standard rules (leading zeros do not count, trailing zeros after a decimal point do).
    • sig_figs_match: combines the three functions above to decide whether a response is both numerically correct and written to the right number of significant figures. To compare the rounded response and rounded answer, it uses Python's math.ulp function as the allowed error, instead of comparing with ==. Rounding a number to a fixed number of significant figures does not always produce an exact result in floating point arithmetic, so two values that should be treated as equal can end up differing by a tiny amount. math.ulp(x) gives the smallest possible gap between x and the next representable floating point number, so using it as the allowed error accepts only that unavoidable floating point rounding error, without being loose enough to accept a genuinely different value. This is the same idea as numpy.spacing, which is what the equivalent Is-Similar feature uses, but math.ulp is part of the standard library, so no new dependency (numpy) needed to be added to this project.
  • app/evaluation.py: added the sig_figs parameter, with significant_figures accepted as an alternative name. Added checks that raise an error if sig_figs is used together with atol/rtol, or if sig_figs is not a positive whole number.
  • app/context/symbolic.py: in check_equality, added a check at the very start of the function. If sig_figs is set and the comparison is a direct response = answer (not some other custom comparison), the function now uses sig_figs_match instead of its normal equality check. This has to happen before the normal check, not after, because 92.00 and 92 would otherwise already be treated as equal (they are the same number) before the significant figures were ever counted.
  • app/context/physical_quantity.py: made the equivalent change inside quantity_match, which is the function that decides whether a response matches the answer for physical quantities. When sig_figs is set, it replaces the normal value check with sig_figs_match, using the response's value as originally written (before unit conversion) so the significant figures can be counted correctly. The unit comparison itself is unchanged. Also updated the existing logic that estimates a tolerance from the number of significant figures in the answer, so it is skipped whenever sig_figs is set, keeping the two features independent.
  • app/docs/user.md: added a section explaining the new sig_figs/significant_figures parameter, with an example and the significant figure counting rules.
  • app/docs/dev.md: added a short technical note for developers on how the feature fits into the existing code.
  • app/tests/expression_utilities_test.py: added tests for the four new functions.
  • app/tests/symbolic_evaluation_test.py and app/tests/physical_quantity_evaluation_test.py: added tests that call the full evaluation function with sig_figs set, covering correct answers, wrong values, wrong precision, non-numeric answers, and the new error cases.

Checklist

  • Tests added/updated
  • Docs updated (app/docs.md) if user-facing behaviour changed

Closes #270

@peterbjohnson peterbjohnson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions:

  • units: are there tests for physical quantities with different units, e.g. 1 mile vs 1609 m, combined with sig_figs? How does it work?

  • symbols: can symbols be used with sig_figs? E.g. x**2?

  • relaxed: we previously discussed two versions of sig_figs. One is the scientific version, implemented here, the other is the more relaxed sense of 'correct within x sig_fig'. Are we offering both, or only the first? What's our reasoning?

  • clashes: is this the first time we've had incompatible parameters? Any thoughts on user experience, given that we provide atol/rtol by default?

I've also made a minor inline comment

Comment thread app/context/symbolic.py
# must still fail, so this can't be gated behind "ordinary equality already returned False".
lhs_string = criterion.children[0].content_string().strip()
rhs_string = criterion.children[1].content_string().strip()
if {lhs_string, rhs_string} == {"response", "answer"}:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this different to the test in the physical_quantity context line 277, which permitted the opposite variant too (both of response==answer and answer==response)?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a different way of testing the same thing. Pytho sets denoted by {} are unordered, and this will check if both sets are equal, no matter the order, so lhs_string could be equal to answer or response and as long as rhs_string` is set to the other one, the if statement condition will be true.

@m-messer

Copy link
Copy Markdown
Member Author

Questions:

units: are there tests for physical quantities with different units, e.g. 1 mile vs 1609 m, combined with sig_figs? How does it work?
  • Thats a great question, I'll add some unit tests now.

    symbols: can symbols be used with sig_figs? E.g. x**2?

  • Not at the moment, only responses that can be parsed as numbers recongised in Python, this exlcudes pi and e. Do we want to support symbols?

    relaxed: we previously discussed two versions of sig_figs. One is the scientific version, implemented here, the other is the more relaxed sense of 'correct within x sig_fig'. Are we offering both, or only the first? What's our reasoning?

  • Currently, we only implement the scientific version. I will ask Phil about this when I meet with him later today.

    clashes: is this the first time we've had incompatible parameters? Any thoughts on user experience, given that we provide atol/rtol by default?

  • It is yes. I think this needs further UI design. Currently, we will return an error if both are used; however, that would only appear if compareExpressions is tested, it might be something we report as part of the parameters in info.

@peterbjohnson

Copy link
Copy Markdown
Member

Ok. I suggest don't develop content in the platform that uses sig_fig until we discuss further with Phil. Items to carry forward:

  • different interpretations of sig_fig (tolerance or scientific)
  • clashing parameters
  • use with symbols

If for now you prefer to merge so that it keeps up with developments, that's fine (I can't see the accept option in my current environment), as long as the above points are attended to.

I noticed there's conflicts but I presume that's fixable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Support Sig Fig

2 participants